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□  Since  the  1960s,  the  military  and  much  of 
the  civilian  business  community  have  used  a 
systematic  approach  to  curriculum  develop¬ 
ment.  This  approach,  often  called  Instruc¬ 
tional  Systems  Development  (ISD),  grew  out 
of  the  "systems  analysis"  approach  to  prob¬ 
lem  solving  that  became  popular  after  World 
Wtrll  (I&arslty,  1984).  Since  then,  hundreds 
of  manuals  have  been  published  which  pro¬ 
vide  instruction  on  how  to  design  and  develop 
instruction  (Andrews  &  Goodson,  1980;  Mon- 
temerlo  &  Tennyson,  1976).  Although  these 
manuals  differ  in  many  ways,  thty  all  share 
the  same  basic  components  (Dick  &  Carey, 
1985).  The  systems  jroach  to  instructional 
development  is  oriented  toward  job  perfor¬ 
mance  and  concerned  with  the  identification 
of  tiaiiur.g  requirer,-  =nts  based  on  job  data  ob¬ 
tained  from  the  fieid.  i  rom  this  job  analysis, 
learning  objectives  ar'  formulated  with  cor¬ 
responding  tests  which  are  used  to  assess 
trainees'  progress  toward  meeting  the  objec¬ 
tives.  ISD  procedures  aLo  attempt  to  prescribe 
optimal  insfructional  strategies,  media,  and 
sequencing  for  various  types  of  training  ac¬ 
tivities.  In  short,  ISD  is  an  attempt  to  ensure 
that  training  is  job  relevant,  cost  effective,  and 
efficient  (Montague  &  Wulfeck,  1986). 

Over  the  past  fifteen  years,  the  Navy  has 
moved  steadily  toward  more  and  more  de¬ 
tailed  ISD  guidelines.  These  very  specific  pro¬ 
cedures  are  almost  "cookbook"  in  nature  and 
are  designed  for  the  relatively  inexperienced 
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curriculum  developer.  While  many  of  the 
Navy's  coiuses  are  originally  developed  by 
contractors  (who  also  follow  the  ISD  guide¬ 
lines),  the  task  of  revising  and  managing  the 
courses  foils  to  Navy  persoimel  assigned  to 
the  various  training  commands.  The  Navy 
teaches  over  7,000  courses  and  has  within  its 
training  commands  a  limited  number  of  ci¬ 
vilian  education  specialists  who  monitor  the 
development  and  management  of  training. 
Hence,  Navy  instructors  with  subject  matter 
expertise  are  primarily  responsible  for  revis¬ 
ing,  developing,  and  maintaining  training  ma¬ 
terials,  yet  frequently  these  personnel  have 
had  limited  or  no  experience  in  curriculum  de¬ 
velopment.  Thus,  the  intent  of  the  Navy's 
specific  guidelines  is  to  provide  enough  in¬ 
formation  to  allmv  nonexperts  in  training  to 
develop  adequate  instruction  with  minimal 
outside  assistance.  ' 

The  work  reported  in  this  article  is  part  of 
an  ongoing  research  effort  on  quality  control 
of  the  instructional  development  process.  The 
goal  of  this  project  was  to  evaluate  hmv  effec¬ 
tively  the  procedures  and  principles  of  the 
systems  approach  have  been  applied  in  the 
development  and  maintenance  of  Navy  tech¬ 
nical  training;  that  is,  to  determine  whether 
the  Navy's  guidelines  have  resulted  in  the  pro¬ 
vision  of  adequate  instruction. 

BACKGROUND 

One  of  the  military's  first  attempts  to  proced 
uralize  ISD  was  The  Inlerservice  Procedures  for 
Instructional  Systems  Development  (Branson  et 
al.,  1975).  This  manual,  developed  under  con¬ 
tract  by  Honda  State  Umversity  and  published 
by  the  Army  and  Navy,  presented  an  ISD 
model  based  on  five  phases.  Analysis,  Design, 
Development,  Implementation,  and  Control. 
The  intent  of  the  manual  was  to  provide  suf 
fident  guidance  to  nonexperts  in  ISD  to  al 
low  them  to  adequately  design,  develop,  and 
manage  training. 

Since  1975  there  have  been  additional  man 
uals  providing  even  more  detailed  guidance 
on  the  implementation  of  ISD.  For  example, 
the  Handbook  for  Training  Materials  Development 
{DOD-HNDK-292)  (Department  of  the  Navy,  1986) 
provides  a  very  rigid  set  of  procedures  in  a 


500-page  two-volume  manual.  There  are  over 
50  complex,  multipage  flowcharts  that  guide 
the  user  through  the  ISD  process.  This  is  just 
one  of  several  documents  currently  used  in 
the  military  to  guide  the  development  process. 
In  a  study  conducted  in  the  rate  1970s,  Vine- 
beig  and  Joyner  (1980)  reviewed  the  ISD  meth¬ 
odologies  and  practices  in  the  Army,  Navy, 
Marine  Corps,  and  Air  Force.  Thqr  evaluated 
57  courses  at  33  different  organizations  to  de¬ 
termine  the  effectiveness  of  the  various  ISD 
procedures  and  their  application  within  the 
four  services.  They  found  that  the  guidelines 
or  ISD  manuals  did  not  ensure  adequate  im¬ 
plementation  of  ISD.  Often  training  did  not 
meet  job  requirements:  training  objectives  of¬ 
ten  ignored  task  requirements  and  tests  were 
developed  without  regard  to  traming  objec¬ 
tives.  In  feet,  Vineberg  and  Joyner  found  that 
there  was  evidence  to  suggest  that  "objectives 
are  often  prepared  after  the  fact  and  are  de¬ 
rived  from  training  content  rather  than  used 
to  generate  it."  (page  xii).  This  practice  runs 
directly  counter  to  the  ISD  process.  Similar 
problems  have  been  found  in  other  broad  re¬ 
views  (Hodak,  Middleton,  &  Rankin,  1979; 
Middleton,  Zajkowski,  &  Aagard,  1979)  as  well 
as  in  evaluations  of  specific  courses  (Ellis,  1985; 
Lockhart,  Sturges,  VanMatre,  &Zachai.  1979; 
Montague,  Ellis,  &  Wulfeck,  1983;  Stem  & 
Fredericks,  1982). 

The  work  reported  in  this  article  builds  on 
these  studies.  An  extensive  evaluation  of  Navy 
classroom  training  was  conducted  to  deter- 
mme  how  effectively  the  systems  approach  is 
being  applied  within  the  Navy.  (Note  that  dif¬ 
ferent  branches  of  the  Navy  use  somewhat  dif¬ 
ferent  development  guidelines  and  manuals.) 
The  study  included  evaluation  of  segments  of 
approximately  one  week's  duration  from  100 
courses  for  enlisted  personnel  that  ranged 
from  the  basic  entry-level  course  to  the  more 
sophisticated  advanced  training  course.  For 
each  course,  a  representative  sample  of  the 
course  objectives,  test  items,  and  instructional 
presentations  were  reviewed.  The  evaluation 
was  of  classroom  training  rather  than  labora 
tory  mstruction,  and  the  focus  was  on  objec 
hves  that  were  asscaa.ed  with  written  tests 
rather  than  performance  tests.  Objechves,  test 
items,  and  presentabons  were  evaluated  to  de¬ 
termine  the  following: 
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1.  appropriateness  of  the  objectives,  test 
items,  and  presentation  for  the  overall 
course  training  goal  (which  is  based  on  the 
job  requirements); 

2.  consistency  of  objectives  and  test  items  (do 
die  test  items  match  the  objectives?); 

3.  adequacy  of  the  test  items  (are  the  test 
items  properiy  constructed?); 

4.  consistency  of  the  classrcrom  presentation 
and  the  training  objectives  (does  the  pre¬ 
sentation  contain  the  right  combination  of 
presentation  components?); 

5.  adequacyoftheclassroompresentation(is 
the  information  presented  clearly  stated 
and  job  oriented?); 

6.  instructor  and  mstructionaledectiveness. 

The  Course  Evaluation  System  (CES)  (Ellis, 
Knirk,  laylor,  &  McDonald,  1987)  was  the  pri¬ 
mary  tool  for  evaluating  the  100  Navy  courses. 
This  evaluation  model  is  based  on  research, 
techniques,  and  procedures  developed  by 
Bloom  (1984);  Ellis  (1985);  EUis,  Wulfock,  and 
Fredericks  (1979);  MeniU  (1983);  Merrill,  Reige- 
luth,  and  Faust  (1979);  and  Montague,  Ellis, 
and  Wulfock  (1983).  The  basic  CES  method¬ 
ology  has  been  empirically  validated  for  va¬ 
lidity  and  reliability  in  a  number  of  research 
and  practical  applications  (e.g.,  Ellis,  1985;  El¬ 
lis,  Wulfock,  Merrill,  Wood,  &  Baker,  1979; 
Montague,  Ellis,  &  Wulfock,  1983;  Stem  & 
Fredericks,  1982;  Taylor,  Ellis,  &  Baldwin 
1988).  While  the  CES  has  been  validated,  it 
should  be  noted  that  it  is  designed  to  be  ap¬ 
plied  to  objectives-based  training  that  has  been 
developed  using  the  ISD  process.  Further,  the 
CES  evaluation  criteria  were  distilled  from  the 
work  dted  above  and  the  explidt  goals  con¬ 
tained  in  many  ISD  models  (e.g.,  Branson  et 
al.,  1975).  The  following  section  provides  de¬ 
tails  on  the  CES  and  how  it  was  applied. 


METHOD 

Course  Sampling 

One  hundred  enlisted  Navy  courses  were  se- 
leded  for  evaluation.  Courses  were  selected 
to  be  a  representative  distribution  of  the  total 
number  of  enlisted  training  courses  by  war- 
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fore  community  (surfors,  air,  and  subsurfoce) 
as  well  as  type  of  school  (apprentice,  ad¬ 
vanced,  and  fleet).  There  were  some  con¬ 
straints  in  the  sampling  process.  In  order  to 
minimize  travel,  most  of  the  courses  selected 
were  conducted  in  the  San  Diego  area,  al¬ 
though  a  variey  of  commands  and  squadrons 
were  induded.  A  total  of  16  commands  and 
squadrons,  three  outside  the  San  Diego  area, 
were  induded  in  the  evaluation. 


TTie  Course  Evaluation  System  Process 

Because  of  time  constraints,  entire  courses 
could  not  be  evaluated.  So,  after  an  initial  in- 
tervierv  with  the  course  instructors  and  man¬ 
agers,  a  representative  sample  of  each  course 
(approximately  one  week  of  instruction)  was 
selected  for  evaluation,  based  on  the  recom¬ 
mendations  of  the  course  personnel.  The 
evaluation  of  the  objectives,  test  items,  and 
classroom  presentations  was  a  six-step  process. 

Step  2;  Classify  Ohjeclives 

Qassifying  objectives  makes  them  easier  to 
evaluate  for  appropriateness  and  to  match 
with  test  items.  It  also  fodlitates  the  evalua¬ 
tion  of  the  instructional  presentation.  An  ob¬ 
jectives  listing  was  obtained  for  each  course 
and  the  objectives  related  to  the  sample  of  in¬ 
struction  selected  for  evaluation  were  dassi- 
fi“d.  The  CES  has  a  dassification  scheme  that 
is  a  simplified  version  of  those  developed  for 
Merrill's  Component  Display  Theory  (CDT) 
(Merrill,  1983)  and  the  Navy's  Instructional 
Quality  Inventory  (IQI)  (Montague,  Ellis,  & 
Wulfock,  1983).  These  schemes  were  simpli¬ 
fied  to  be  better  aligned  with  the  type  of  train¬ 
ing  occurring  in  Navy  classrooms  and  to 
focilitate  the  dassification  process. 

In  the  CES  there  are  three  categories  of  ob¬ 
jectives  that  are  assodated  with  written  (pa¬ 
per  and  pencil)  tests:  Remember,  Locate,  and 
Mental  Skills.  Remember  objectives  require  the 
student  to  recall  or  recogrrize  specific  informa¬ 
tion,  such  as  basic  facts,  steps  of  procedures, 
or  formulas  for  mathematical  computations. 
For  Locate  objectives,  the  student  must  be  able 
to  find  information  m  technical  manuals  or  m- 
strucbons.  The  student  would  not  be  required 
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to  remember  equipment  specifications  or 
maintenance  procedures,  but  would  need  to 
know  how  to  locate  those  specifications  and 
procedures  in  a  technical  manual.  For  Mental 
Stalls  objectives,  the  student  must  solve  prob¬ 
lems  using  rules,  concepts,  and  principles.  For 
example,  a  student  might  be  required  to  clas¬ 
sify  a  series  of  sonar  scope  displays,  solve 
mathematical  problems,  or  predict  the  effect 
of  fidd  contamination  on  aircraft  petfermancc. 

The  relationship  of  these  three  categories 
of  objectives  to  the  task  and  content  catego¬ 
ries  from  CDT  and  the  IQI  is  as  follows:  the 
Remember  category  corresponds  to  remember- 
fect,  procedure,  concept,  and  principle  cate¬ 
gories;  the  Locate  category  corresponds  to  the 
use-procedure  category;  and  the  Mental  Skills 
category  corresponds  to  the  use-principle  and 
use<oncept  categories.  This  entire  scheme  is 
described  in  detail  in  the  CES. 

Step  2:  Match  Objectives  and  Test  Items 

Test  items  were  obtained  for  the  objectives 
classified.  Test  items  were  taken  from  course 
module  and  end-of-course  exams.  Items 
from  mstructor-developed  quizzes  were  not 
included. 

Learning  objectives  are  made  up  of  three 
main  components:  the  conditions  imder  which 
the  task  is  to  be  performed,  the  action  or 
behavior  that  the  trainee  is  to  exhibit,  and 
the  standard  by  which  performance  is  to  be 
judged.  To  match  objectives  and  test  items, 
each  test  item  was  evaluated  to  see  whether 
the  conditions,  action,  and  standard  specified 
by  the  objective  were  incorporated  in  the  test 
item  or  items  associated  with  it.  In  other 
words,  were  the  students  being  tested  the  way 
the  objective  stated  they  would  be  tested? 

For  Remember  objectives,  the  actions  and 
conditions  had  to  be  an  exact  match.  That  is, 
if  the  objective  required  the  student  to  be  able 
to  recall  from  memory,  then  a  corresponding 
multiple-choree  test  item  would  not  match  be¬ 
cause  It  reqmres  only  that  the  student  be  able 
to  recognize  the  correct  answer.  For  example, 
if  the  objective  is  that  the  student  "state  the 
defmition  of  Minimum  Force  as  it  appears  m 
the  Master-At-Anns  Manual,"  and  it  has  a  cor¬ 


responding  multiple-choice  test  item,  then  the 
actions  and  conditions  do  not  match.  The  ac¬ 
tions  do  not  match  because  the  objective  re¬ 
quires  the  student  to  state  the  definition  or 
recall  it  from  memory,  but  the  test  item  allcnvs 
the  student  to  select  the  correct  definition.  The 
conditions  (givens)  do  not  match  because  the 
test  item  provides  a  list  of  possible  answers 
that  are  not  specified  in  the  objective. 

Finally,  the  number  of  objectives  for  which 
there  were  no  matching  test  item(s)  were  re¬ 
corded.  According  to  most  ISD  guidelines, 
there  should  be  at  least  one  test  item  for  each 
objective.  Note  that  in  many  courses  not  all 
objecti^'es  are  actually  tested  because  of  time 
constraints.  However,  if  there  was  a  test  item 
(usually  in  a  test  item  bank)  for  an  untested 
objective,  it  was  counted  as  a  match. 

Step  3:  Identify  Course  Training  Goal 

After  objectives  were  classified  and  linked 
to  test  items,  one  or  more  senior  instructors 
were  interviewed  at  the  school  to  determine 
the  overall  course  training  goal  so  that  judg¬ 
ments  could  be  made  as  to  objective  and  test 
appropriateness.  Instructors  were  asked  to 
identify  the  course  training  goal  from  the 
following: 

•  Heavily  Supervised  On-the-Job  Training 
(OJT).  The  student  should  be  generally  fa¬ 
miliar  with  terminology,  technical  docu¬ 
mentation,  and  duties  required  on  the  job. 

•  Minimally  Supervised  OJT.  The  student 
should  receive  enough  hands-on  training 
so  that  upon  graduation  he  or  she  will  be 
able  to  perform  the  job  with  very  little 
supervision. 

•  Skilled  Performer.  The  student  should  re¬ 
ceive  enough  hands-on  training  so  that 
upon  graduation  from  the  course,  he  or  she 
will  be  able  to  perform  the  task  with  no  as¬ 
sistance  from  a  supervisor. 

Clearly,  the  course  training  goal  has  impor¬ 
tant  impLcabons  for  course  objectives  and  test 
items.  Objectives  and  test  items  should  con 
tam  more  ngorous  requirements  as  one  moves 
from  the  Heavily  Supervised  OJT  to  the  Min¬ 
imally  Supervised  and  Skilled  Performer  levels 
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Step  4:  Determine  Appropriateness  of 
Each  Objective 

The  conditions,  action,  and  standard  of  each 
learning  objective  wrere  rated  for  appropri¬ 
ateness  for  the  course  training  goal  and  future 
job  requirements.  AH  ratings  were  "yes"  or 
"no"  judgments.  An  objective  is  considered 
to  be  inappropriate  in  any  one  of  its  parts — 
conditions,  action,  or  standard — ^is  inappro¬ 
priate.  An  example  of  an  objective  that  would 
be  rated  as  inappropriate  is  one  that  requires 
students  to  memorize  information  or  proce¬ 
dures  from  technical  or  reference  manuals 
used  on  the  job.  In  this  case  the  action  is  in¬ 
appropriate  because  the  student  is  required 
to  memorize  information  rather  than  locate  it, 
and  the  conditions  is  inappropriate  because  stu¬ 
dents  are  not  given  the  technical  or  reference 
manual  in  the  classroom. 

A  second  criterion  for  rating  objective  ap¬ 
propriateness  concerns  the  memory  require¬ 
ments  of  the  job.  If  critical  information  must 
be  memorized  for  on-the-job  performance,  and 
the  objective  requires  only  that  the  student  rec¬ 
ognize  that  information,  then  the  conditions 
and  actions  would  be  rated  as  inappropriate. 
For  example,  the  objective  "The  student  will  be 
able  to  identify  from  a  list  the  safety  precau¬ 
tions  to  be  observed  when  performing  main¬ 
tenance  on  the  aircraft"  would  be  rated  as 
inappropriate.  The  action  is  inappropriate  be¬ 
cause  the  student  should  be  required  to  state 
the  precautions  from  memory,  and  the  condi¬ 
tion  is  inappropriate  because  the  student  is 
given  a  list  of  possible  answers  from  which  to 
choose,  which  would  never  happen  on  the  job. 

A  third  reason  for  an  inappropriate  rating 
concerns  the  objective  standards.  For  exam¬ 
ple,  if  the  course  traming  goal  was  identified 
as  Skilled  Performer  but  the  standard  spea- 
fied  in  the  objectives  for  course  tests  was  75%, 
then  the  objective' s  standard  would  be  rated 
as  inappropriate — a  75%  standard  is  too  low 
to  guarantee  a  person  could  perform  duties 
at  the  Skilled  Performer  level. 

In  addition  to  rating  each  objective  for  ap¬ 
propriateness,  at  least  two  course  instructors 
were  asked  to  rate  each  objective  as  to  its  im¬ 
portance  to  the  job  the  students  were  bemg 


trained  to  do.  Instructors  were  asked  to  rate 
each  objective  as  "essential,"  "nice  to  know," 
or  "nonessential"  while  keeping  the  job  re¬ 
quirements  in  mind. 

Step  5:  Determine  Appropriateness  and 
Adequacy  of  Test  Items 

Here  appropriateness  refers  to  how  well  the 
test  item  conditions,  action,  and  standard 
match  the  overall  course  goal  as  determined 
in  Step  3.  The  procedure  and  criteria  for  rat¬ 
ing  test  item  appropriateness  were  the  same 
as  those  for  rating  objective  appropriateness. 

Test  item  adequacy  refers  to  how  well  the 
item  is  constructed;  that  is,  were  the  guide¬ 
lines  for  test  construction  adhered  to?  For  ex¬ 
ample,  for  a  fill-in-the-blank  item,  is  it  worded 
so  that  only  one  word  or  phrase  could  cor¬ 
rectly  complete  the  sentence?  Are  there  gram¬ 
matical  or  other  cues  to  the  correct  answer? 
Is  there  only  one  blank  in  a  single  item?  For 
multiple-choice  tests,  are  all  choices  believ¬ 
able?  For  a  complete  set  of  adequacy  criteria, 
see  the  CES  (Ellis  et  al.,  1987),  Handbook  for 
Testing  in  Navy  Schools  (Ellis  &  Wulfeck,  1982), 
and  Ellis  &Wulfeck  (1986). 

Step  6:  Evaluate  Classroom  Presentation 

At  least  30  minutes  of  classroom  instruction 
associated  with  knowledge-type  objectives 
(Remember,  Locate,  Mental  Skills)  for  each  of 
the  100  courses  were  observed.  First,  the  con¬ 
sistency  and  the  adequacy  of  the  presentation 
were  evaluated  using  checklists  from  the  CES 
(Ellis  etal.,  1987). 

In  order  for  a  presentation  to  be  consistent, 
it  must  contain  an  appropriate  combination 
of  presentation  parts  or  components  required 
for  the  type  of  objective  of  the  traimng.  There 
are  four  possible  components — statements, 
practice  remembenng  with  feedback,  exam¬ 
ples,  and  practice  using  with  feedback — de- 
saibed  as  follows: 

1.  Statements  present  essential  information 
needed  b)  the  student  to  master  the  ob¬ 
jective.  For  example,  a  statement  for  a  Re¬ 
member  objective  that  requues  the  student 
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to  know  the  part  names  of  a  lathe  could 
involve  the  instructor  naming  the  parts  on 
a  diagram  or  on  the  actual  equipment  in 
hont  of  the  dass.  For  a  Locate  objective  that 
requires  the  student  to  find  information  in 
a  manual,  the  instructor  should  state  or  vis¬ 
ually  present  the  steps  of  the  procedure  one 
goes  through  to  find  things  in  a  manual. 

2.  Practice  remembering  vnth  feedback  provides 
an  opportunity  for  the  students  test  their 
ability  by  recalling  or  recognizing  specific 
information  presented  in  the  statement. 

3.  Examples  demonstrate  the  task  to  be  per- 
fonned.  For  example,  the  instructor  might 
demonstrate  looking  things  up  in  a  man¬ 
ual  or  fiJling  out  paperwork. 

4.  Practice  using  mth  feedback  ^ves  students  an 
opportunity  to  perform  the  task  and  receive 
corrective  feedback. 

Not  all  presentation  components  are  re¬ 
quired  for  each  type  of  objective.  Table  1  pro¬ 
vides  the  components  required  for  the  three 
types  of  knowledge  objectives. 

Adequacy  refers  to  a  number  of  design  prin¬ 
ciples  that  promote  student  learning.  In  rat¬ 
ing  the  adequacy  of  the  presentations,  the 
organization  and  format  of  materials  were 
evaluated  as  well  as  whether  additional  expla¬ 
nation  was  provided  tc  help  students  learn 
and  retain  the  information.  Specifically,  each 
presentation  component  was  rated  for  being: 

1.  Separated.  Is  the  component  separated  from 
the  rest  of  the  instruction?  Students  need 


to  be  able  to  find  important  information. 

2.  Identified.  Is  the  component  identified 
so  that  students  know  what  they  are  look¬ 
ing  at? 

3.  Clearly  slated.  Is  the  component  clearly 
stated  so  that  the  student  can  under¬ 
stand  it? 

Presentations  were  also  rated  for  the  pres¬ 
ence  or  absence  of  "helps" — ^Does  the  com¬ 
ponent  contain  someihing  to  help  the  student 
better  understand? — and  for  being  "job  ori¬ 
ented" — ^Is  the  component  oriented  toward  job 
performance? 

To  summarize,  a  consistent  presentation  con¬ 
tains  the  presentation  components  required 
for  the  type  of  objective(s),  and  an  adequate 
presentation  presents  the  component  informa¬ 
tion  in  the  most  effective  and  efficient  maimer. 

After  the  consistency  and  adequacy  evalu¬ 
ation,  the  overall  effectiveness  of  the  class¬ 
room  presentation  was  evaluated  using  a  CES 
checklist  that  rated  (1)  instructor  behavior,  (2) 
media  and  materials,  (3)  the  classroom  envi¬ 
ronment,  and  (4)  student  behavior  in  the 
classroom.  The  overall  effectiveness  checklist 
contained  a  series  of  yes/no  questions  related 
to  presentation  effectiveness.  For  example,  in¬ 
structors  were  evaluated  as  to  whether  they 
(1)  motivated  students  by  explaining  why  the 
content  should  be  learned,  (2)  told  students 
how  they  would  be  tested,  (3)  monitored  stu¬ 
dents'  progress,  and  (4)  ensured  students  were 
actively  involved. 


1ABLE1  □  Presentation  Components  Required  for  Each  Objective  Type 


REQUIRED  PRESENTAnON  COMPONENTS 


OBJECTIVE  TYPE 

Statement 

Practice  Remembering 
with  Feedback 

Examples 

Practice  Using 
tvith  Feedback 

Remember 

Required 

Required 

Not  required 

Not  required 

Locate 

Required  if  no 
recent 
Remember 
objective 

Required  if  no 
recent 

Remember 

objective 

Required 

Required 

Mental  Skills 

Required  if  no 
recent 
Remember 
objective  or 
ciid 

Required  if  no 
recent 

Remember 
objective  or 
aid 

Required  with 
variety  of 
examples 

Required  with 
variety  of 
practice  problems 
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RESUUS  AND  DISCUSSION 

Objedive  Classification 

Of  the  1,945  knowledge  objectives  classified, 
46  percent  were  Remember  objectives  (n  = 
888),  52  percent  were  Locate  objectives  (n  = 
1,004),  and  2  percent  were  Mental  Skills  ob¬ 
jectives  (n  =  53).  The  low  percentage  of  Men¬ 
ial  Skills  Ol^’ectives  is  consistent  with  a  previous 
study  that  classified  over  30,000  objectives 
(Wetzel,  VanKekeiix,  &  Wulfeck,  1987).  This 
finding  reflects  the  routine  procedural  nature 
of  the  majority  of  Navy  jobs. 

Objective  and  Test  Item  Evaluation 

Table  2  provides  the  results  of  the  objective 
and  test  item  evaluation.  A  total  of  1,945  ob¬ 
jectives  and  2,734  test  items  were  evaluated. 

Objective  Not  Appropriate 

An  objective  was  considered  to  be  inappro¬ 
priate  if  any  one  of  its  parts  (conditions,  ac¬ 
tion,  or  standard)  was  inappropriate.  Table  3 
shows  that  over  half  (56  percent)  of  the  objec¬ 
tives  rated  were  inappropriate.  While  some 
objective  conditions  and  actions  were  found 
to  be  inappropriate  for  the  reasons  described 
in  the  Method  section  of  this  article,  the  ma¬ 
jority  of  objectives  were  rated  as  inappropri¬ 
ate  because  of  problems  with  the  standard. 
Table  3  shows  that  51  percent  of  the  objectives 
did  not  have  appropnate  standards.  This  was 
because  often  standards  were  too  low  to  sup¬ 
port  the  course  training  goal.  The  cause  of  this 
is  that  many  of  the  Navy  documents  that 


1ABLE2  □  Objective  and  Test  Item  Evaluation 
Results 


Objectives  {n  =  1,945) 


Objective  Not  Appropriate 

56% 

Conditions  Not  Appropriate 

14% 

Standards  Not  Appropriate 

51% 

Actions  Not  Appropriate 

8% 

Objective  Not  Essential 

8% 

Objective  Not  Tested 

49% 

Test  Items  (n  =  2,734) 


Test  Item  Does  Not  Match  Objective 

48% 

Conditions  Do  Not  Match 

38% 

Standards  Do  Not  Match 

6% 

Actions  Do  Not  Match 

41% 

Test  Item  Not  Adequate 

7% 

Test  Item  Not  Appropriate 

38% 

Note:  Inappropriate  objectives  or  test  items  may  have  more 
than  one  inappropriate  part. 

guide  the  development  of  instruction  and  tests 
arbitrarily  set  passing  standards.  For  exam¬ 
ple,  the  Training  Specifications  Manual  (NAMT- 
GlNSr.Pi540.2G)  specifies  that  a  passing  grade 
on  a  written  test  should  be  between  63  and 
75%  (although  it  does  allow  for  individual  test 
items  to  require  100%  accuracy).  When  applied 
to  courses  with  a  training  goal  at  the  Mini¬ 
mally  Supervised  level,  this  standard  was 
considered  too  low  because  the  Minimally 
Supervised  job  incumbent  needs  a  higher  level 
of  job  knowledge  to  perform  effectively.  This 
problem  occurred  in  over  30  of  the  courses 
reviewed. 

Table  3  highlights  this  relationship  between 
training  goals  and  inappropriate  objectives.  A 
significant  difference  was  found  among  train¬ 
ing  goals  for  the  frequency  of  inappropriate 
objectives  (x^  =  311.50,  p  <  .01).  The  higher 
the  training  goal,  the  more  likely  that  the  ob- 


TABLE3  □  Inappropriate  Objective  Components  by  Training  Goal 


INAPPROPRIATE  OBJECTIVE  COMPONENT 

TRAINING  GOAL 

Standard 

Action 

Condition 

Skilled  Performer 
n  =  262 

S7% 

13% 

34% 

Minimally  Supervised 
n  =  1063 

70% 

10% 

9% 

Heavily  Supervised 
«  =  620 

3% 

5% 

15% 

100 
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jective  standard  would  be  rated  as  inappro¬ 
priate.  One  possible  explanation  for  this  is  that 
often  the  minimum  standard  established  by 
training  development  documents  command 
becomes  the  otdy  standard.  Test  standards  are 
not  always  adjusted  upward  to  meet  the  de¬ 
mands  of  courses  with  higher  training  goals. 

Objective  Not  Essential 

A  veiy  low  percentage  of  objectives  were  rated 
by  instructors  as  nonessential.  Most  of  the  ob¬ 
jectives  were  identified  as  either  essential  or 
"nice  to  know."  In  other  words,  the  instructors 
felt  that  most  of  the  objectives  should  be  re¬ 
tained  because  they  were  either  necessary 
or  provided  motivational  information,to  the 
student. 

Objective  Not  Tested 

Almost  half  (49  percent)  of  all  objectives  that 
we  reviewed  were  not  tested,  even  though  the 
vast  majority  of  them  were  considered  essen¬ 
tial  by  the  instructors.  In  some  cases  this  oc¬ 
curred  because  the  courses  had  a  large  number 
of  very  specific  objectives.  In  these  courses  it 
was  common  to  review  a  one-hour  lesson  that 
had  over  a  dozen  enabling  or  supporting  ob¬ 
jectives.  For  example,  when  a  terminal  (main) 
objective  for  a  lesson  required  the  student  to 
locate  equipment  information  in  a  manual,  sev¬ 
eral  enabling  objectives  required  the  student 
to  describe  individual  dials  and  knobs.  Time 
constraints  made  it  impossible  for  all  of  these 
objectives  to  be  tested.  However,  even  courses 
with  fewer  and  more  broadly  stated  objectives 
had  large  percentages  of  untested  objectives. 
Further,  even  when  a  broadly  stated  objective 
had  an  associated  test  item,  it  often  related 
only  to  a  portion  of  the  objective. 

Test  Item  Does  Not  Match  Objective 

Almost  half  (45  percent)  of  all  test  items  did 
not  match  the  corresponding  objective.  In  other 
words,  at  least  one  objective  component  (ac¬ 
tion,  condition,  or  standard)  did  not  match 
the  related  test  item.  In  most  cases,  when  the 
conditions  and  actions  did  not  naatch,  the  ob¬ 


jective  required  the  student  to  recall  the  cor¬ 
rect  answer  and  the  test  item  required  the 
student  to  recognize  the  correct  answer. 

There  were  also  a  number  of  courses  in 
which  the  test  item  standards  did  not  match 
related  objective  standards.  This  usually  oc¬ 
curred  when  the  objective  stated  that  a  stu¬ 
dent  must  correctly  answer  a  certain  number 
of  test  items  corresponding  to  that  objective. 
Objective  and  test  standards  did  not  match 
because  test  items  for  individual  objectives 
were  not  scored  separately.  That  is,  a  student 
could  pass  the  test  based  on  the  total  score 
even  though  he  or  she  failed  to  answer  the 
correct  number  of  test  items  related  to  an  in¬ 
dividual  objective. 

Test  Item  Not  Adequate  or  Not  Appropriate 

Table  2  also  presents  the  findings  for  inad¬ 
equate  and  inappropriate  test  items.  There 
were  very  few  inadequate  test  items  (although 
the  fact  that  the  researchers  were  not  subject 
matter  experts  and  therefore  were  unable  to 
judge  whether  the  alternative  choices  for 
multiple-choice  items  were  realistic  could  have 
resulted  in  less  items  being  classified  as  in¬ 
adequate). 

Over  a  third  of  the  test  items  were  rated  as 
inappropriate.  The  most  frequent  reason  for 
this  was  that  the  standard  was  considered  too 
low  to  support  the  training  goal  of  the  course. 
This  problem  is  the  result  of  inappropriate 
standards  for  the  related  objectives. 


Classroom  Presentation  Evaluation 

Presentation  Consistency 

Most  of  the  presentations  associated  with 
knowledge  objectives  were  consistent.  In  other 
words,  they  contained  the  presentation  com¬ 
ponents  required  for  the  particular  type  of  ob¬ 
jective.  However,  we  found  that  in  almost  half 
(46  percent)  of  the  presentations  associated 
with  Remember  objectives,  the  "practice  re- 
membenng  with  feedback"  component  was 
either  incomplete  or  absent  altogether  Fur¬ 
ther,  in  a  quarter  of  the  presentations  for  Lo¬ 
cate  objectives,  both  the  statement  and  the 
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"practice  remembering  with  feedback"  com¬ 
ponents  were  either  incomplete  or  absent. 

Presentation  Adequacy 

In  general,  the  presentations  observed  were 
adequate.  The  majority  of  the  objective  com¬ 
ponents  met  the  adequacy  criteria.  Approxi¬ 
mately  67  percent  of  them  were  adequate  on 
the  separated,  identified,  and  clearly  stated 
criteria.  These  criteria  provide  students  with 
information  about  where  they  are  in  the  in¬ 
structional  process.  This  type  of  "road  map" 
is  important  and  can  afiect  student  learning 
and  retention.  In  addition,  over  70  percent  of 
the  components  contained  helps,  and  over  80 
percent  of  them  were  rated  as  job  oriented. 

Effective  Checklist  Results 

The  majority  of  the  instructors  we  observed 
delivered  the  instruction  in  an  effective  man¬ 
ner  and  actively  involved  the  students  in  the 
learning  process  by  using  proper  questioning 
techniques  (75  percent  of  the  courses  ob¬ 
served)  and  checldng  student  comprehension 
(75  percent  of  the  courses  observed).  However, 
the  evaluation  revealed  that  many  instruc¬ 
tional  practices  that  have  been  shown  to  have 
a  positive  effect  on  student  learning  in  the  ci¬ 
vilian  environment  are  not  being  utilized 
within  military  training.  These  include  peer 
instruction  (11  percent  of  the  courses  ob¬ 
served),  advance  organizers  (34  percent  of  the 
courses  observed),  and  use  of  external  rewards 
(6  percent  of  the  courses  observed). 

As  is  probably  the  case  with  most  civilian 
and  military  courses,  the  typical  classrooms 
observed  during  the  evaluations  could  have 
been  classrooms  of  25  years  ago.  The  media 
used  almost  always  consisted  of  chalkboards 
and  occasionally  overhead  transparencies  of 
tables  or  diagrams  firom  a  technical  manual. 
The  benefits  of  modem  trainmg  technology 
have  not  reached  the  Navy  classroom. 

GENERAL  DISCUSSION 

In  summary,  for  the  objective  and  test  item 
evaluation  it  was  found  that  (1)  a  majority  (56 
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percent)  of  the  1,945  knowledge  objectives  ex¬ 
amined  were  inappropriate  for  the  course 
training  goal  and  future  job  requirements;  (2) 
almost  half  (49  percent)  of  the  objectives  were 
not  tested;  (3)  about  half  (48  percent)  of  all  test 
items  did  not  match  related  objectives;  (4)  over 
a  third  (38  percent)  of  all  test  items  were  con¬ 
sidered  inappropriate;  and  (5)  the  majority 
of  test  items  (93  percent)  were  adequately 
constructed. 

For  the  presentation  evaluation  it  was  found 
that  (1)  delivery  of  most  presentations  was  ad¬ 
equate  and  consistent  with  the  type  of  objec¬ 
tive  being  taught;  (2)  practice  for  almost  half 
of  the  presentations  was  incomplete  or  not 
present;  (3)  many  instructional  strategies 
proven  to  be  effective  in  civilian  classrooms 
were  not  utilized;  and  (4)  use  of  media  in  the 
classroom  was  generally  limited  to  chalk¬ 
boards  and  overhead  transparencies. 

The  problems  identified  in  this  study  are 
not  unique  to  the  military.  Any  time  one  at¬ 
tempts  to  implement  a  process  as  complex  as 
ISD,  there  are  potential  problems.  Many  of 
the  problems  found  were  primarily  due  to  ob¬ 
jectives  and  tests  that  were  written  without 
re^d  to  the  level  of  knowledge  required  to 
support  future  job  requirements.  This  ac¬ 
counts  for  the  rather  overwhelming  number 
of  objectives  and  test  items  that  were  classi¬ 
fied  as  inappropriate.  The  problem  is  easy  to 
recognize,  but  may  be  difficult  to  fix.  The  no¬ 
tion  of  training  goals  and  related  passing  test 
scores  is  not  foreign  to  those  who  develop  in¬ 
struction.  However,  there  is  an  institutional 
bias  toward  univeral  passing  scores  and  mul¬ 
tiple-choice  test  items  that  is  extremely  diffi¬ 
cult  to  overcome.  This  is  true  in  civilian  as  well 
as  military  education  and  training.  Nickerson 
(1989)  recently  recognized  this  problem,  but 
J.,0  emphasized  that  the  solution  is  not  easy 
or,  in  some  cases  (e.g.,  mental  skills,  think¬ 
ing),  not  obvious.  As  a  first  step  in  military 
training  development,  ISD  guidelines  should 
eliminate  the  notion  of  global  passing  scores 
that  are  applied  across  large  numbers  of 
courses,  and  instead  let  passing  scores  be  set 
for  each  individual  course  based  on  job  re¬ 
quirements. 

The  problems  with  untested  objectives  and 
with  objectives  that  are  not  well  matched  to 
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test  items  may  be  difficult  to  solve  for  the  same 
reasons.  Here,  too,  as  a  first  step  the  instruc¬ 
tional  development  guidelines  should  be  re¬ 
vised  to  promote  the  development  of  more 
job-related  objectives  and  test  items.  There 
should  be  less  emphasis  on  rote  memoriza¬ 
tion  and  more  emphasis  on  compreherrsion 
and  £amiliariza^''>n  with  technical  documen- 
jtion.  Guidelines  should  provide  instruction 
on  how  to  write  objectives  and  test  items  to 
ensure  that  all  objectives  get  tested  anJ  that 
the  test  items  are  written  to  match  the  ob¬ 
jectives. 

The  lack  of  sophisticated  media  is  also  a  dif¬ 
ficult  problem.  Most  instructors  used  the 
chalkboard  and  transparencies  containing  in* 
formation  from  technical  manuals.  Many  of 
the  presentations  could  have  been  enhanced 
with  other  media  to  improve  motivation,  in¬ 
crease  attention,  and  provide  better  quality 
(e.g.,  more  accurate)  training.  However,  cur¬ 
rent  budget  limitations  make  it  difficult  to  up¬ 
grade  classrooms.  This  problem  truiy  diminish 
as  all  types  of  media  become  more  affordable. 

Instructional  practices  that  have  been  shown 
to  have  a  positive  effect  on  learning  within  the 
civilian  community,  such  as  peer  instruction 
aind  systematic  assessment  of  student  prog¬ 
ress,  were  not  utilized  in  the  classes  that  were 
observed  in  this  study.  TInese  practices  should 
be  applied  in  Navy  classrooms  on  a  tri?’  ba¬ 
sis  to  see  how  well  they  work.  However,  the 
mechanisms  for  doing  this  are  not  well  estab¬ 
lished,  and  there  is  an  institutiorud  resistance 
to  substantial  changes  in  the  way  instruction 
is  conducted  (Semb,  Ellis,  Montague,  &  Wul- 
feck,  1990). 

One  important  area  in  which  there  were  rel¬ 
atively  few  problems  was  the  classroom  in¬ 
structor.  The  Navy's  instructor  training  and 
subsequent  certification  process  (acaanpUshed 
in  the  schools)  is  producing  highly  qualified 
instructors.  The  instruction  observed  was  well 
presented,  the  instructors  were  enthusiastic, 
and  students  were  attentive.  The  major  defi¬ 
ciency  in  Lire  classrooms  is  the  lack  of  time 
given  to  allow  students  to  practice  what  they 
must  learn.  Investing  in  practice  yields  high 
return,  as  it  increases  proficiency  more  than 
any  other  activity.  Maruigers  should  enstuc 


that  their  curriculum  development  guide¬ 
lines  stress  the  importance  of  practice  in  the 
classroom. 

Implications  for  Instructional  Designers. 

Melvin  Montemerlo  (1979)  pointed  out  that 
one  of  the  dangers  of  proceduralizing  the  ISD 
process  is  that  designers  and  developers  of¬ 
ten  tend  to  lose  sight  of  the  whole  process. 
Course  designers  frequently  focus  on  the  pres¬ 
ent  development  step  without  thinking  about 
the  final  product.  Many  of  the  problems  found 
within  the  Navy's  implementation  of  ISD  sup 
port  this  observation.  Problems  such  as  inap¬ 
propriate  course  standards,  inappropriate 
memorization  re<{uirements,  inadequate  test¬ 
ing,  and  lack  of  practice  cicur  when  the  course 
developer  fails  to  keep  focus  on  the  ultimate 
course  goal:  to  train  students  to  perform  a  spe¬ 
cific  job.  Thus,  standards  get  set  that  fall  to 
meet  job  requirements  and  students  do  not 
practice  enough  to  have  sufficient  job  knowl¬ 
edge  or  to  learn  at  a  level  needed  on  the  job. 
To  develop  effective  instruction,  course  design¬ 
ers  must  focus  continually  on  the  relationship 
between  the  components  of  the  course  (i.e., 
objectives,  test  items,  and  instructional  pre¬ 
sentation)  and  the  requirements  of  the  job  the 
student  will  be  expected  to  perform  upon 
graduation.  □ 
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